MEMORANDUM December 16, 2019 


TO: Alyssa Howell 
Mathematics Director, Office of Secondary Curriculum and Development 


FROM: Carla Stevens 
Assistant Superintendent, Research and Accountability 


SUBJECT: RELATIONSHIP BETWEEN IMPLEMENTATION FIDELITY OF IMAGINE 
MATH® AND HISD SECONDARY STUDENTS’ EDUCATIONAL 
ACHIEVEMENT, 2018-2019 


Imagine Math® is an adaptive, online mathematics program designed to supplement in-class 
instruction. The program is a districtwide initiative aimed at students from grades six through 
twelve. This evaluation report examined the fidelity of implementation of Imagine Math® and the 
associations between its use and student achievement in Houston Independent School District 
(HISD) secondary schools in the 2018-2019 academic year. Comparisons were made between 
middle school and high school levels for implementation fidelity and 2019 STAAR mathematics 
performance levels. 


Key findings include: 

» Imagine Math® was made available to 110,416 students in 26 middle schools, 62 high 
schools, and 15 combined K-12 schools for a total of 103 schools for the 2018-2019 
academic year. 

= Average level of adherence to fidelity components approached expectation at middle schools 
and high schools. 

« Frequency of use of Imagine Math® by teachers and analysis of the Imagine Math® reports 
for instructional purposes was low. 

« Students’ use of the program approached expectation, with high-school students completing 
the highest number of lessons followed by middle-school students. 

« The most important predictor of middle school and high school students’ 2019 academic 
achievement was their previous year’s STAAR scale scores. 

» The number of Imagine Math® lessons attempted with fidelity was a statistically significant 
predictor of student achievement; the more lessons attempted, the better the STAAR scaled 
score. 


Further distribution of this report is at your discretion. Should you have any further questions, 
please contact me at 713-556-6700. 
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Relationship Between Implementation Fidelity of Imagine Math® and HISD Secondary Students’ Educational 


Achievement, 2018-2019 
Prepared by Georgia Graham 


Abstract 


Imagine Math® is a standards-aligned, adaptive, interactive online mathematics program designed to supplement in- 
class instruction for students. This evaluation report examined the fidelity of implementation of Imagine Math® and 
the associations between its use and student achievement in Houston Independent School District (HISD) secondary 
schools in the 2018-2019 academic year. Based on teacher results, the research showed that the average levels of 
adherence to fidelity components were approaching expectations for both middle schools and high schools. At the 
teacher-level, frequency of use of Imagine Math® by teachers and analysis of the Imagine Math® reports for 
instructional purposes was low. Nevertheless, student progress and students’ use of the program, were approaching 
expectations. Results of multiple linear regression analyses showed that, at the teacher-level, the Data Analysis 
component was a consistent Statistically significant predictor of student achievement across levels. At the student- 
level, however, the most important predictor of students’ 2019 academic achievement was their previous year’s 
STAAR scale score. The number of Imagine Math® lessons attempted with fidelity was also a Statistically significant 
positive predictor of student achievement; the more lessons attempted the better the STAAR scaled score. 


Background 

Imagine Math® is an adaptive, online mathematics 
program designed to supplement in-class instruction. 
Imagine Math®, the program, formerly known as Think 
Through Math (TTM), was funded in 2013-2014 through 
the Texas Education Agency’s Texas Success Initiative 
and was free to students in grades three through eight. 
Additional services may have been purchased by 
campuses to serve students enrolled in high school 
Algebra I courses (HISD, 2014). In 2017, HISD re- 
launched the program, as “a districtwide initiative aimed 
at students from grades six through twelve who exhibited 
below satisfactory performance on state assessments” 
(HISD, 2018, p.12). 

Imagine Math® supports students and teachers with 
personalized, interactive math instruction through a 
research-based and _— standards-aligned curriculum 
(Imagine Learning, 2019). The program combines virtual 
and on-site teacher training, live teacher support, unique 
student motivation and contests, and adaptive instruction 
in a web-based learning system to help students learn 
math. Students are immersed in a_ language-rich 
curriculum that uses data to scaffold concepts for each 
learner, ultimately leading to deep understanding and 
college and career-readiness. Since the system is 
adaptive, students learn in their zone of proximal 


development with the right degree of challenge (Imagine 
Learning, 2019). 


Introduction 

The Imagine Math® model is guided by five basic 
tenets andeight classroom practices that provide 
guidelines for program training and implementation that 
lead to desired outcomes (Imagine Learning, 2019). The 
Imagine Math® website outlines the five basic 
pedogeological tenets of the model as being: (i) 
personalized learning driven by the Quantile® 
Framework; (ii) on-demand tutoring by certified math 
teachers; (iii) productive struggle with proactive 
intervention; (iv) math tools; and (v) a motivation system 
that develops confident thinkers. Research has supported 
these tenets as being key elements to improve student 
learning (An, 2004; Wang et al. 2015). The 
interrelationship between the program features is built on 
the Quantile® Framework for Mathematics. This 
framework is a universal scale for both assessment and 
instruction that is used for interpreting student 
performance (Lennon & Burdick, 2004; Williamson, 
2006). 

The classroom best practices for successful 
implementation can be summed up in eight student- 
focused and _ teacher-focused strategies (Imagine 
Learning, 2019). The three student-focused strategies 


cover student usage: (i) students need to have dedicated 
computer time (lab schedule or computer cart schedule); 
(ii) 60-90 minutes of use (cumulative) per week; (iii) 
and, complete and pass 2—3 lessons per week or roughly 
30 lessons a year to increase their scores (Imagine 
Learning, 2019). Each of Imagine Math® default grade 
level pathways consist of roughly 30+ lessons; 
hence, Think 30. This standard is largely grounded in 
research done between 1991 and 1999 with 632 junior 
high and high school students learning 9" grade algebra 
using a similar product like TTM for an average of 15— 
20 contact hours during the academic year (Meyer, 
Steuck, Miller, & Kretschmer, 2000). 

The remaining best practices are teacher-focused, 
three of which are based on teacher strategies: (iv) a 
motivation plan, extra incentives and rewards for the 
class, is outlined and explained to students (i.e. pizza 
party); (v) students are encouraged to use Imagine Math® 
Notebook/Journal; and (vi) teachers should be reviewing 
these regularly. There are two best practices related to 
data usage: (vii) teachers are expected to review student 
data regularly; and (viii) conduct regular consultation 
with students regarding individual progress in Imagine 
Math®. Finally, teachers are expected to encourage 
students to use Math Help where available in the program 
(Imagine Learning, 2019). 


Literature Review 

The induction of technology in the classroom is 
premised on the belief that increased access and use of 
computers and digital tools will enhance teaching and 
learning outcomes, increase efficacy, and improve 
development of critical skills amongst students (Brown, 
& Warschauer, 2006; Fried, 2008; Keengwe, Onchwari, 
& Wachira, 2008; Sandholtz, Ringstaff, & Dwyer,1999). 
The use of innovative web-based technology, especially 
in the assessment phase, enhances students’ learning in 
mathematics when the web-based assessment system is 
well designed and used correctly and in a timely manner 
(Nguyen, Hsieh, & Allen, 2006). 

The focus here is not on the design of the system, but 
rather on the implementation of the intervention 
according to the program developers’ design. “Fidelity of 
implementation is traditionally defined as_ the 
determination of how well an _ intervention is 
implemented in comparison with the original program 
design during an efficacy and/or effectiveness study” 
(O’Donell, 2008, p. 33). This is the extent to which the 
intervention corresponds to the originally intended 
program (i.e. adherence, compliance, integrity). 


Factors that affect implementation fidelity 

The literature highlights several factors that impede 
or facilitate implementation ffidelity: teacher 
characteristics, classroom characteristics, and school 
characteristics. A cursory scan of literature on 
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implementation of technology-based programs in schools 
identified a variation in technology usage across and 
between campuses, which was attributed to differences in 
implementation related to teacher roles, administration, 
professional development opportunities, and availability 
of supplementary program supports (Bebell & O’Dwyer, 
2010; Shapley, Sheehan, Maloney, & Caranikas-Walker, 
2010; Turnbull, 2002). The _ responsibility of 
implementing any school-based intervention falls 
squarely on the shoulders of the teachers; therefore, 
teacher characteristics are an important factor in 
implementation fidelity. According to Bebell & Kay 
(2010), it is impossible to exaggerate the influence of 
individual teachers in the success or failure of 
technology-based learning models because “teachers 
nearly always control how and when students access and 
use technology during the school day” (p. 47). Therefore, 
teacher buy-in for technology immersion is key to 
successful implementation (Shapley et al., 2010). 

Research on teacher buy-in has identified that 
teachers are more likely to support school improvement 
projects when adequate training and resources are 
available, and support is provided by program developers 
and staff (Turnbull, 2002; Shapley et al., 2010). In 
looking at the implementation of a comprehensive school 
reform in relation to teacher buy-in, Turnbull (2002) 
found that teacher participation in the selection of a 
program was not a significant indicator for successful 
implementation. Rather, predictors of immediate and 
long-term buy-in included training and _ support, 
administration buy-in, and control over classroom 
implementation. Administrator buy-in was also identified 
by Bebell & Kay (2010) as an indicator of successful 
implementation. In looking at the implementation of 
educational technology in middle school environments, it 
was found that it was important to ensure that school 
administrators were on-board and believed that the 
program would be beneficial for the school. The variation 
in usage between schools of educational technology 
impacted the outcomes (Bebell & Kay, 2010). 

In addition to teacher factors, research has also 
focused on _ classroom factors that increased 
implementation fidelity and the sustainability of quality 
program implementation by teachers (Hamre, Downer, 
Jamil, & Pianta, 2012). In a study of methods of 
implementation fidelity, McKenna, Flower, & Ciullo 
(2014) found that giving teachers control over how 
programs were being implemented in classrooms, 
allowed them to decide what changes were needed and 
how those changes are made without compromising 
fidelity of implementation. When fidelity is adequate, but 
student performance is lagging, teachers can reconceive 
and adjust the interventions to more adequately meet 
student needs (McKenna, Flower, & Ciullo, 2014). To 
ensure fidelity, teachers should be provided with 
additional coaching and feedback on _ intervention 


delivery, comprehensive training and support (Darling- 
Hammond & Richardson, 2009). 

Finally, it is important to take into consideration 
school characteristics when exploring implementation 
fidelity (McNamara & Hollinger, 2000). In a study that 
looked at school characteristics and student performance, 
Heck & Mayor (1993) defined school characteristics as 
more than administration buy-in. It included socio- 
economic profile of schools, campus leadership, teacher 
practices and behavior, extracurricular activities, and 
resources, both human and material, available at schools. 
What they found was that differences among schools 
were attributed to background characteristics of students 
and the demographic variables of the schools that they 
attended (Heck & Mayor, 1993). Another study 
concluded that the relationship between school 
environmental characteristics (e.g., student background, 
school size, school level) and school academic indicators 
(e.g., attitudes toward knowledge and achievement, staff 
professional development) were useful in explaining the 
types of outcomes that were produced (Berg & Cornell, 
2016). 


Conceptualizing implementation fidelity 

Drawing on existing literature and the program logic 
model, linkages are built between activities and outcomes 
(Emshoff, 2008). Implementation fidelity is 
conceptualized as the extent that the intervention has 
been implemented as designed based on the components 
of the intervention (Caroll et al., 2007; Dane & 
Schneider, 1998; Dusenbury, Brannigan, Falco, & 
Hansen, 2003; Mihalic, 2004). Imagine Math® has 
identified five implementation components (Formative 
Assessment, Data Analysis, Leadership, Teacher Usage, 
and Student Progress). The literature has highlighted 
Teacher Buy-in and Training and Support as additional 
components of fidelity (Table 1). These six components 
are then divided into the subcategories of adherence or 
moderators (Caroll et al., 2007). Adherence deals 
specifically with implementation fidelity, whether the 
intervention is being implemented as intended, while 
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moderators refer to the examination of factors that impact 
the degree of fidelity (Caroll et al., 2007). 


Research Questions 

Based on HISD’s prior exposure to TTM (now 
Imagine Math®), the Secondary Curriculum and 
Development Department at HISD made the Imagine 
Math® program available to all students in the district 
beginning in the 2017—2018 academic year. With this 
investment, it is essential to determine the impact of this 
web-based program on student performance, how these 
outcomes vary based on demographic characteristics, and 
elements of the implementation process that has helped 
to improve these outcomes. As such, the research 
questions are as follows: 


1. To what extent was Imagine Math® implemented in 
HISD secondary schools? 


2. What hindered or facilitated the level of fidelity of 
Imagine Math® in HISD secondary schools? 


3. What were key factors that predicted Imagine Math® 
student performance on the 2019 STAAR Math 3-8 
and Algebra 1 EOC exams? 


Districts have access to a variety of web-based 
assessment and practice resources to improve students’ 
academic achievement. However, poor implementation 
can adversely affect effectiveness. That is not to say that 
an inadequate response to an intervention could not also 
be a mismatch between the practice and the students’ 
needs. It, then, becomes essential for districts to 
determine the degree to which they implement evidence- 
based practices as intended to determine if an inadequate 
student response is due to poor implementation or 
inappropriate selection of intervention. With increased 
investment in web-based resources, this study was 
designed to investigate the impact of the Imagine Math® 
program on secondary student performance on state-level 
assessments. 


Components Measures 
Adherence—The degree that variables are | Data Analysis Use of program reports to inform teaching 
implemented is the degree of Data Management Ability to access and retrieve reports 
implementation fidelity Frequency of Use Time on site 

Leadership (MLE) Leadership support of program 

Student Progress Number of lessons attempted 
Moderator —Variables that affect level of | Training and Support Teacher received training and support 
achieved implementation Teacher Buy-in Teacher perception of program 


Identification of essential components 
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Method 

For the 2018-2019 academic year, Imagine Math® 
was made available to 110,416 students in 26 middle 
schools, 62 high schools, and 15 combined K-12 schools; 
for a total of 103 schools, including charter schools. 


Sample 

The sample consisted of teachers who completed the 
online survey and their associated students which were 
linked through the Teacher Appraisal and Development 
System (TADS). A total of 247 teacher surveys were 
received, only 196 were completed and included in this 
analysis. The sample of survey respondents consisted of 
55 teachers from middle schools, 128 from high schools, 
and 13 from schools with combined grades ranging from 
K-12. Survey respondents had an average of 9.2 years of 
teaching experience for middle schools (SD=8.9), 7.4 
years of teaching experience for high schools (SD=7.1), 
and 9.1 years of teaching experience for combined 
schools (SD=6.5). The mean length of time using 
Imagine Math® for middle school teachers was 2.9 years 
(SD=1.3), 2.6 years for high school teachers (SD=1.3), 
and 2.7 years for teachers at combined schools (SD=.95). 

The student sample was derived from the classroom 
of teacher survey respondents. The sample excludes 
students who were absent from the previous year’s State 
of Texas Assessments of Academic Readiness (STAAR) 
3-8 or STAAR End-of-Course (EOC) exams, given that 
previous year’s mathematics performance constituted a 
baseline control. Only first-time test takers were 
included. Middle school students who took Algebra I 
were not included in the sample as the initiative was 
aimed at low performing students. Student demographic 
data were drawn from the Public Education Information 
Management System (PEIMS) Fall Resubmission files. 
The final analytic sample consisted of 44,592 students, of 
these 24,587 students (55.1%) had greater than zero 
lessons completed, while 8,907 (31.7%) of these students 
used the math software with fidelity. Most of the 
students, 53.0 percent (n=23,626) were enrolled in a high 
school, 38.2 percent (n=17,025) were enrolled in a 
middle school, and 8.8 percent (n=3,941) attended a 
combined K-12 school (Appendix—A, p. 12). 

The ethnic composition of the sample was mainly 
Hispanic, with 57.0 percent at middle schools, 64.4 
percent at high schools, and 51.5 percent at combined 
schools. Students identified as economically 
disadvantaged comprised 81.5 percent of the overall 
sample, 84.1 percent at middle schools, 81.0 percent at 
high schools, and 73.7 percent at combined schools. 
More than half of the students were at risk of dropping 
out of school. Specifically, 60.5 percent of middle school 
students, 62.8 percent of high school students, and 54.8 
percent of students at a combined school were at-risk. 
Gifted and talented students made up 17.3 percent of the 
overall sample, with 17.3 percent at middle schools, 18.4 
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percent at high schools, and 10.5 percent at combined 
schools. 


Data Collection 

Teacher Survey. The teachers completed an online 
survey based on the Imagine Math® Implementation 
Rubric (2018). The rubric included benchmarks relating 
to teacher data usage, professional development, and 
report usage. The survey included measures of 
adherence, with teachers rating their level of agreement 
with statements on a 5-point Likert scale ranging from 
strongly agree (4) to strongly disagree (0) for two 
indicators: data usage (4 items) and leadership (3 items). 
Teachers used a 4—point Likert scale to rate their 
frequency of use of Imagine Math® reports to guide 
student intervention: 0 (never review) to 4 (more than 
once a week). 

The survey also included measures related to teacher 
buy-in and training and support. The survey was 
disseminated from March 5—April 15, 2019, through 
several online platforms. Imagine Math® posted a link to 
the survey on the dashboard of district teachers, the 
survey was posted on the HISD HUB, and distributed via 
SurveyMonkey. Weekly email reminders were sent out 
to teachers, all secondary school principals, and campus 
math chairs. An academic memo was posted on HISD 
website announcing the survey. Each of the teacher-level 
components were coded as ‘1’ for teachers who used 
Imagine Math® and ‘0’ for those who did not. 

Academic Achievement. The academic outcome 
measures were the 2019 spring administration of the 
STAAR mathematics for grades 6 to 8 and the Algebra 1 
EOC exam results for grades 9 tol2. The STAAR is a 
state-mandated criterion-referenced assessment that 
annually measures students’ academic performance and 
achievement. Student scale scores on the 2019 STAAR 
spring administration was included in the study. Re- 
testers were not included in the study due to over 
exposure to both the program and outcome measures. 
Students’ STAAR data used in this study were retrieved 
from Cognos, a data querying software. Outcomes were 
measured by comparing 2019 HISD student STAAR 
scores with those who met the vendor recommended 
usage level (fidelity) versus those who did not. Imagine 
Math® identifies fidelity of usage as the completion of 
two or more lessons per week, which would mean an 
average of 30+ lessons in the 2018-2019 academic year, 
Think 30. Based on the degree of fidelity, students who 
used the program with fidelity were coded as ‘1’, without 
fidelity coded as ‘2’ and those who did not use coded as 
‘0’. 


Measures 

Student progress is measured using the number of 
lessons attempted variable, according to Imagine Math 
number of lessons attempted is the number of lessons 


completed. Implementation fidelity was measured as the 
level in which adherence to Imagine Math® components 
and related moderators achieved the expected ideal level 
based on the implementation rubric (Appendix—B, p. 13). 
This involved gathering data on Imagine Math® 
components for each of the treatment groups and 
comparing school-to-school variations with the 
expectation of ‘full’ fidelity. 

Adapting a process developed by Shapley, Sheehan, 
Maloney, & Caranikas-Walker (2010), the value for each 
component was computed relative to the maximum value 
of 4, which denotes full fidelity. The scales were 
standardized to allow comparisons across different 
components. A mean fidelity standard score was 
computed on the 0 to 4 scale for each indicator. Z scores 
were calculated to facilitate the grouping of several 
Likert-type items into a ‘survey scale’ using the factor 
analysis technique, Cronbach’s alpha (a), to provide 
evidence that the components of the scale were 
sufficiently intercorrelated and that the grouped items 
measured the underlying variable (Sullivan & Artino, 
2013). The Likert scales were normally distributed, and 
Cronbach's alpha showed reliability level for all fidelity 
components. The results of Cronbach's alpha ranged from 
.707 to .917, denoting a rating ranging from acceptable to 
excellent (Appendix—C, pp.14—15). 


Data Analyses 

Two approaches to data analyses were used. First, 
campus-level analyses used descriptive statistics (counts, 
mean, standard deviation, percentages) to describe how 
the Imagine Math® model and its components were 
implemented. This was followed by using mean standard 
scores to examine the relationship between 
implementation levels and_ student academic 
achievement. The second approach of analyses involved 
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Frequency of Use, Leadership, and Student Progress) and 
moderators (Training and Support, and Teacher Buy-in) 
that supported fidelity. The dependent variable was 
students’ performance on 2019 STAAR mathematics or 
Algebra | EOC exams. Students included in the analyses 
had a prior year STAAR mathematics score. Analyses 
were conducted separately for students in middle school 
and high school. The histogram and scatter plots of the 
residual indicated the assumption of normality, linearity, 
and homoscedasticity were all satisfied. 

Students’ 2019 STAAR scores were regressed on 
their previous years’ STAAR scores, demographic 
characteristics, and Student Progress. Stage two 
investigated the association between treatment fidelity 
indicators and students’ academic outcomes, which were 
regressed on students’ previous years’ STAAR scores, 
demographic characteristics, Student Progress, 
Frequency of Use, Data Analysis, and Data Management 
scores. 

RESULTS 


To what extent was Imagine Math® implemented in 
HISD secondary schools? 


Figure 1 displays the mean fidelity scores by component 
and school level. Mean standard scores for fidelity 
components generally showed to be approaching 
expectation at the middle and high school levels. The two 
moderators, Teacher Buy-in and Training and Support, 
were also approaching expectation. Fidelity scores for 
Teacher Buy-in at middle school (2.4), high school (2.4), 
and combined school (2.0) levels showed _ that 
respondents found Imagine Math® useful to them and 
their students. Results showed that the mean standard 
score for Training and Support was highest for high 
schools (2.4), followed by middle schools (2.1), and 
combined schools (2.0). Training and Support was 
provided by the vendor and HISD staff designated as the 


Figure 1. Mean Level of Implementation Expectation by School Level 


predictors that measured adherence to fidelity 

components (Data Analysis, Data Management, 
Frequency of Use 
Data Analysis 


Data Management 
Leadership 

Teacher Buy-in 
Training and Support 


Student Progress 


=Combined ms High School 


= Middle School 


Note: Progress towards fidelity was measured at three levels below, 0-1.99; approaching, 2.00—2.99; and met, 3.00-4.99. 
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Imagine Math® lead who assisted with implementation 
and offered support for product usage. 

Leadership had the highest mean standard score 
across campus levels, reflecting approaching expectation 
at the high school level (2.8), middle school level (2.5), 
and at the combined school level (2.6). Respondents 
agreed that there was support for the use of Imagine 
Math® in the classroom by leadership, teachers were able 
to determine how to use the program in the classroom, 
and students had access to computers. Data Management 
was also approaching expectation with a mean standard 
score that ranged from 2.3 at the combined school level 
to 2.8 at the high school level. Based on the mean 
standard scores across campus levels, teachers agreed 
they had the ability to use the Imagine Math® program to 
retrieve reports and manipulate the program. 

Adherence for Data Analysis and Frequency of Use 
was below expectation. The mean standards score for 
Data Analysis was the lowest at the high school level 
(1.0) and middle school level (1.0), compared to the 
combined school level (1.5). This showed that generally, 
survey respondents did not analyze data from Imagine 
Math® to identify needs of low standards and struggling 
students, apply an intervention or lesson according to the 
results, or adjust the program according to student needs. 

The Frequency of Use was also below expectation at 
all three campus levels, high school (1.1), middle school 
(1.0), and combined school (.4), which showed limited 
use of the Imagine Math® program and reports. This was 
also evident by the number of times teachers logged onto 
the Imagine Math® website. Having the highest mean 
standard score for Frequency of Use, high school teachers 
had the highest number of logins in the school year on the 
Imagine Math® website (m=38.1) and spent on average 
4.7 hours on the website. Middle school teachers who had 
a mean log in of 33.4 times in the school year spent an 
average of 3.3 hours on the website. While combined 
schools had a mean log in of 14.7 times in the school year 
and spent an average of 1.2 hours on the website. 

Student Progress measured the strength of adherence 
to the use of Imagine Math® at the student level. The 
mean standard score for the number of lessons students 
attempted was approaching expectation at the high 
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school level (2.7), combined school level (2.5), and 
middle school level (2.0). During the 2018-2019 
academic year, Imagine Math® student usage data 
showed that students at middle schools had the lowest 
number of lessons attempted but, on average, attempted 
486 problems, earned 6,714 points and used the software 
for 4.0 hours. High school students attempted, on 
average, 467 mathematics problems, earned 6,672 points, 
and used the software for 3.6 hours. Students who 
attended combined schools had the second highest score 
for Student Progress and only attempted, on average, 336 
problems, earned 5,323 points, and used the software for 
2.0 hours. 

Table 2 displays the correlations among the seven 
components of fidelity, with statistically significant 
coefficients marked with an asterisk. Overall, there was a 
statistically significant association for six of the seven 
components. The association between Data Management 
and Data Analysis was statistically significant (r=.50). 
There was also a statistically significant association 
between Leadership support of teachers’ use of Imagine 
Math® with Data Analysis (7=.38) and Data Management 
(r=.73). Teacher Buy-in to the use of Imagine Math® was 
associated at a statistically significant level with Data 
Use to inform instruction (7=.39), the ability to manage 
the software to assess student learning (7=.70), leadership 
encouragement and support for the use of Imagine Math® 
in the schools (r=.70), use of the software by teachers 
(r=.47). Training and Support showed the strongest 
possible agreement across all associations (r=.75). 
Student Progress was only positively associated with 
how frequently teachers used the software (7=.29). 


What hindered or facilitated the level of fidelity 
of Imagine Math in HISD secondary schools? 


Training and support for Imagine Math® was 
provided by the vendor as well as district staff designated 
to guide product usage. The vendor provided support 
through one-on-one training, printed material (Quick 
Guide), professional learning events, and planning 
sessions. Figure 2 shows the level of fidelity relative to 
the four elements of Training and Support Quick Guide, 
One-on-one Training, Campus Lead, and Customer 

Success Mangers. 


Data ; Data - vor ‘Frequency of Training and Teacher Student 
Analysis Management Usage Support Buy-in Progress 
|Data Analysis 1.00** | 
‘Data M anagement 30 1.00 
Leadership =| 38" | 73** =~ | .00 | 
\Frequency of Usage | .63** 59 ** 67 ** 1.00 ‘| 
Training and Support| .47 ** 71 72 ** sae* | 100 | 
‘Teacher Buy-in | .39** =| 70** =| .70** are | 75 | 100 | 
‘Student Progress | 1S | 14 |B 2% | or | 08 1.00 
Note: * p<.05  **p<.01 ***p<.000 
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Figure 2. Level of Fidelity for Elements of the Training and 
Support Component by Mean Fidelity Score 
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On average, teachers reported a level of agreement 
with the benefits of the training and support for effective 
implementation as approaching a level of fidelity across 
all elements for each campus level. Teachers agreed that 
the Quick Guide was a helpful resource for setting up and 
using Imagine Math at the middle school level (m=2.5), 
high school level (m=2.4), and combined school level 
(m=2.3). Teachers also reported that the Customer 
Success Managers provided ongoing training and support 
for middle schools (m=2.1), high schools (m=2.5), and 
combined schools (m=2.6). 

Imagine Math® was designed as a support not only 
for students, but teachers as well. Teachers were asked 
about their perception of Imagine Math® as being a 
useful tool for themselves and their students. This was 
measured using five elements that comprised the Teacher 
Buy-in component (Figure 3). Comparisons across 
campus levels indicated that teachers, on average, found 
Imagine Math® to be beneficial to their students and a 
useful instructional support (scores ranged from 2.3 to 
2.8). 


Figure 3. Level of Fidelity for Elements of the Teacher Buy- 
In Component by Mean Fidelity Score 
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On average, teachers approached fidelity relative to 
frequency of use, with teachers at combined schools 
having the highest mean standard score (2.5), followed 
by high schools (2.0), and middle schools (1.7) (Figure 
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4). For fidelity of use, Imagine Math recommends that 
reports are accessed weekly. Results show that teachers, 
on average, used Imagine Math reports monthly. 
Frequency of use of Student Progress reports, on average, 
was approaching expectation at the middle school and 
combined school levels (2.0), and below expectation at 
the high school level (1.4). Teachers also reported that 
they used Imagine Math® to inform instruction on an as 
needed basis, for the most part. With teachers at the high 
school and combined school levels approaching 
expectation for instructional use and below expectation at 
the middle school level. 


Figure 4. Level of Fidelity for Elements of the Frequency of 
Use Component by Mean Fidelity Score 
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The Imagine Math reports are designed to give 
teachers the information they need to help with student 
learning and ensure program success. To do so, teachers 
need to be able to access and retrieve the reports. 
Teachers’ level of agreement with Data Management 
components assessed their ability to retrieve individual or 
group reports, identify areas where they need to adjust 
their instruction based on the reports, retrieve reports that 
best meet their needs, and manipulate the program to 
meet the math needs of individual learners (Figure 5). 


Figure 5. Level of Fidelity for Elements of the Data 
Management Component by Mean Fidelity Score 
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While teachers approached expectation on most of 
the elements of the Data Management component, 
manipulating the Imagine Math® program had the lowest 
mean fidelity score. However, teachers at the high school 
level showed the highest fidelity level related to 
manipulating the Imagine Math program (2.5), followed 
by middle schools (2.0). Teachers at the combined school 
level tended to fall below expectation (1.7). 

The Imagine Math® reports provide information that 
supported teachers in identifying and addressing the 
needs of low standards and struggling students. Based on 
the data, teachers should be adjusting the software 
(groupings, settings, usage frequency, instructional 
pathway, or journaling), applying intervention and 
extension lesson(s), and helping students create goal(s) 
(Figure 6). The level of fidelity for all four elements of 
the Data Analysis component were below expectation, 
with one exception. Teachers were below fidelity in using 
Imagine Math to identify and address the needs of low 
standards and struggling students, with the lowest level 
of fidelity achieved by teachers at the middle school level 
(m=1.1), followed by high schools (m=1.2), and 
combined schools, which approached expectation 
(m=2.0). Additionally, teachers’ responses were below 
expectation relative to applying intervention and 
extension lessons based on the data retrieved. Again, the 
lowest level of fidelity was at middle schools (m=1.0), 
followed by high schools (m=1.1), and then combined 
schools (m=1.7). 


Figure 6. Level of Fidelity for Elements of the Data Analysis 
Component by Mean Fidelity Score 
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What were key factors that predicted Imagine 
Math® student performance on the 2019 STAAR 
Math 3-8 and Algebra 1 EOC exams? 


The research examined key factors to predict the 
performance of HISD students, who were exposed to 
Imagine Math® on the 2019 STAAR Math 3-8 and 
Algebra 1 EOC exams. Correlations between the Imagine 
Math® components were reported in Table 2 (p. 6) and 
the regression statistics in Appendix D, Table D1 to D2 
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(pp 16-17). A two-stage hierarchical multiple linear 
regression was conducted to determine the extent to 
which students’ 2018 STAAR math scale scores and 
demographic characteristics predicted the 2019 STAAR 
math scale scores for middle school and high school 
students (Model 1). The second model investigated 
whether teacher-level components (Data Analysis, Data 
Management, Frequency of Use, and Student Progress) 
predicted higher 2019 STAAR 3-8 math scale scores, 
after controlling for prior score, demographic 
characteristics, and number of Imagine Math® lessons 
completed. 

The hierarchal multiple regression for middle school 
students, model 1, revealed that prior performance, being 
GT, at risk for school dropout, or Black contributed 
significantly to the regression model (p<.000) and 
accounted for 71.1 percent of the variation in 2019 
STAAR 3-8 scale scores. Introducing the teacher level 
variables explained an additional .7 percent of the 
variation in 2019 STAAR 3-8 scale scores. The change 
in R? was significant (p <.000). Combined, the increase in 
student level and teacher level variables accounted for 
71.8 percent of the variance in 2019 STAAR 3-8 scale 
score. The most important predictor of the 2019 STAAR 
3-8 scale score was the previous years’ scale score, 
which uniquely explained 70 percent of the variance in 
2019 STAAR 3-8 scale score. 

In terms of demographic characteristics, being Black 
was a Significant predictor of the 2019 STAAR 3-8 math 
scale score (p<.01). The score of Black students 
decreased by 25.67 scale score points compared to White 
students. For students at risk of dropping out of school, 
the performance was 22.91 scale score points lower than 
for students not at-risk of dropping out (p<.000). As 
expected, the value on the 2019 STAAR 3-8 scale scores 
for GT students was 39.24 scale score points higher than 
for non-GT students (p<.000). 

At the teacher level, Data Analysis and Frequency of 
Use were significant negative predictors of student 
performance. The predicted value of the scale score for 
students whose teacher analyzed the data from Imagine 
Math® was 20.42 scale score points lower than for 
students whose teacher received lower ratings on the 
Data Analysis component (p<.05). Similarly, students 
whose teacher frequently used the program resulted in 
26.16 scale score points lower on 2019 STAAR 3-8 
compared to those whose teachers did not use the 
program (p<.000). 

Model | of the regression for high school students 
predicted 41.5 percent of the variance in the model 
(p<.000). The inclusion of teacher level variables, model 
2, explained an additional 4.1 percent of the variation in 
students’ 2019 STAAR Algebra I EOC scale scores. The 
change in R? was significant (p<.000). Combined, the 
student level and teacher level variables accounted for 
45.6 percent of the variance in students’ 2019 STAAR 


Algebra I EOC exam scores. For model 1 and model 2, 
the previous years scale scores were the strongest 
predictor of 2019 STAAR Algebra I EOC exam scale 
score. 

At the student-level, gender, at-risk status, GT, and 
home language accounted for a statistically significant 
amount of the variance in both models. Female students 
showed an average increase of 139.96 scale score points 
more than male students (p<.000). For students at risk of 
dropping out of school, the performance was 130.67 scale 
score points lower than for students not at-risk of 
dropping out (p<.000). The score of GT students 
increased by 140.12 scale score points compared to non- 
GT students (p<.01). 

Students who identified their home language as 
Spanish showed an average decrease of 182.5 scale score 
points compared to those who spoke other languages 
(p<.05). This held true as well for students whose home 
language was English, the predicted value of their 
performance was 221.47 scale score points lower than for 
students who spoke other languages (p<.000). 

With the addition of teacher-level variables, high 
school students who attempted thirty or more Imagine 
Math® lessons obtained, on average, an increase of 
270.93 scale score points compared to those students who 
did not complete any lessons (p<.000). High school 
students whose teachers’ fidelity component scores met 
expectations for managing the Imagine Math® website 
saw an increase of 133.33 scale score points compared to 
students whose teachers were below expectation for Data 
Management (p<.000). 

Teacher usage had an inverse effect on student scale 
scores. Teachers who used Imagine Math® at least once a 
month showed a decrease of 49.87 scale score points 
compared to students whose teacher did not use Imagine 
Math® (p<.000). Similarly, the predicted value of scale 
score for students whose teacher analyzed the data from 
Imagine Math® was 91.16 scale score points lower than 
for students whose teacher received lower ratings on the 
Data Analysis component (p<.01). 


Discussion 


The overarching goal of immersing Imagine Math® 
into HISD secondary schools was to increase students’ 
academic achievement as measured by the state 
assessment, STAAR. Therefore, it became important to 
understand whether Imagine Math® was associated with 
higher test scores. The evaluation used the Imagine 
Math® rubric and existing literature to develop a model 
of fidelity based on teacher- and _ student-level 
components. The evaluation examined how fidelity 
components used in the model linked to student 
outcomes. 
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The intention of the evaluation was to assess the 
predictive strength of variables that measured the extent 
to which Imagine Math® was used at the student and 
teacher levels. A two-level multiple linear regression 
model was used to determine if there was an association 
between student academic achievement and the number 
of lessons attempted. Moreover, the analysis sought to 
test the relationship between academic achievement by 
teacher-level components (Data Analysis, Data 
Management, Frequency of Use, and Student Progress). 

It was found that across middle and high school 
levels, schools were approaching expectation on four of 
the seven fidelity components (Data Management, 
Leadership, Training and Support, and Student Progress). 
The strength of the association between these 
components was assessed. At the teacher-level, there was 
a Statistically significant association among the six 
components of fidelity. However, at the student-level, the 
only significant positive association was between 
Frequency of Use by teachers and Student Progress; 
unfortunately, this was the component that had the lowest 
level of implementation fidelity. 

Though there was a bivariate correlation only 
between Frequency of Use by teachers and Student 
Progress, the regression showed an association between 
Data Management and Data Analysis at the high school 
level and Data Analysis and Frequency of Use at the 
middle school level once other variables were controlled 
for. Of the three indicators that showed an association in 
the regression model, only Data Management had a 
positive association. This indicator had one of the highest 
mean level of implementation fidelity for high schools. 
The predicted value of scale score for students whose 
teacher managed the data from Imagine Math® to meet 
student needs was higher than for students whose teacher 
received lower ratings on the Data Management 
component. 

With Data Analysis, at the middle school and high 
school levels, the predicted value of scale score for 
students whose teacher analyzed the data from Imagine 
Math® was lower than for students whose teacher 
received lower ratings on the Data Analysis component. 
Once again, the level of adherence to the Data Analysis 
component was below expectation across campuses. 
Based on the results for the adherence to fidelity 
components, many HISD teachers reported they did not 
use the software with fidelity or conduct the required data 
analysis to adjust the program to meet their students’ 
needs. Imagine Math® recommends that teachers review 
reports on a weekly basis in order to adjust instruction to 
student learning (Imagine Learning, 2019). 

While Frequency of Use showed a significant 
correlation to student performance it had the lowest level 
of fidelity component score across campuses. Though 
low, high school teachers had the highest usage and high 
school students achieved the highest Student Progress 


(Figure 1, p. 5). When included in the regression model, 
Frequency of Use was a statistically significant negative 
predictor of student performance at the middle school 
level. The scale scores that a student would most likely 
achieve if his or her teacher would have used Imagine 
Math® were lower than for students whose teacher 
received lower ratings on the Frequency of Use 
component. Student Progress, the number of Imagine 
Math® lessons attempted, therefore, appears to be 
dependent on how frequently teachers use Imagine 
Math®. 

At the student-level, the student’s prior score 
predicted more of the variance at the middle school 
(70%) than at the high school level (51%). As a result, 
the Imagine Math® program predictors accounted for a 
larger percentage of the variation in student performance 
for high schools compared to middle schools. Being GT 
was also a statistically significant positive predictor of 
student performance at both campus levels. GT students 
who used Imagine Math® had a higher scale score than 
those who were not GT. Students who were at-risk of 
dropping out who used Imagine Math® tended to perform 
lower than their counterparts who were not at risk. 

The number of Imagine Math® lessons attempted 
with fidelity was a consistently positive predictor of 
students’ STAAR mathematics scores. Imagine Math® 
recommends that students attempt at least 30 lessons an 
academic year, Think 30 (Imagine Learning, 2019). The 
predicted value of scale score for students who attempted 
30 or more lessons was higher than for students who 
completed fewer than 30 lessons. While the fidelity 
component for Student Progress, number of lessons 
attempted, was approaching expectation across campus 
levels, this did not align with the amount of time spent on 
the software or problems attempted. The number of 
problems attempted, hours spent on the software, or 
incentives may be key elements; however, students need 
to be encouraged to complete the lessons. 

Teachers become critical to the success or failure of 
any technology-based learning model as they create 
opportunities for students to use technology, either in the 
classroom, at home, or through incentives. The influence 
that teachers may have on effective implementation of 
technology-based learning models could be attributed to 
teachers who use the program frequently as they appear 
to be more likely to create opportunities for students to 
do the same (Bebell & Kay, 2010). 

The findings from this evaluation are consistent with 
previous research that found that the classroom best 
practices of Imagine Math®, when done with fidelity, are 
key elements that appear to contribute to the 
improvement in student learning (Meyer, Steuck, Miller, 
& Kretschmer, 2000). Teacher usage, data analysis, and 
data management, as well as student usage, are essential 
elements of the model. The District should provide 
continued support to increase students’ and teachers’ use 
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of the Imagine Math® program and related reports to 
achieve the level of fidelity that would lead to improved 
student academic performance in mathematics. 
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APPENDIX-A 


Demographic Characteristics of 2018—2019 HISD Secondary Students by School Level 


Overall Sample Middle School. ———sHighSchool_ == Combined 
N % n % n % n % 

Gender Female 20,832 46.7 7,838 46.0 11,066 53.2 1,928 48.9 
Male 23,757 53.3 9,185 54.0 12,560 46.8 2,012 51.1 

Ethnicity Black 10,464 23.5 4,667 27.4 5,076 21.5 721 18.3 
Hispanic 26,958 60.5 9,707 57.0 15,222 64.4 2,029 51.5 

White 3,119 7.0 1,653 9.7 1,897 8.0 501 12.7 

Other 4,051 9.1 998 5.9 1,431 6.1 690 17.5 

Home Language Spanish 21,315 47.8 7,818 45.9 11,331 48.0 2,166 55.0 
English 19,291 43.3 7,399 43.5 10,557 44.7 1,335 33.9 

Other 3,986 8.9 1,808 10.6 1,738 7.4 440 11.2 

Economically No 8,245 18.5 2,713 15.9 4,494 19.0 1,038 26.3 
Disadvantaged Yes 36,347 81.5 14,312 84.1 19,132 81.0 2,903 73.7 
Immigrant No 42,543 95.2 16,125 94.7 22,541 95.4 3,787 96.1 
Yes 2,136 4.8 898 5.3 1,085 4.6 153 3.9 

Homeless No 40,906 91.7 15,594 91.6 21,667 91.7 3,645 92.5 
Yes 3,686 8.3 1,431 8.4 1,959 8.3 296 7.5 

At-Risk No 17,282 38.8 6,717 39.5 8,785 37.2 1,780 45.2 
Yes 27,307 61.2 10,306 60.5 14,841 62.8 2,160 54.8 

Gifted/ Talented No 36,874 82.7 14,074 82.7 19,275 81.6 3,525 89.5 
Yes 7715 17.3 2,949 17.3 4,351 18.4 415 10.5 

SPED No 41,031 92.0 15,724 92.4 21,627 91.5 3,680 93.4 
Yes 3,558 8.0 1,299 7.6 1,999 8.5 260 6.6 


Source: 2018-2019 PEIMS student databases. Percentages may not total 100 due to rounding. 
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APPENDIX-B 


Scoring Rubric for Measuring Fidelity of Implementation of Imagine Math® in HISD Schools 


Hement 


Data Analysis 


Data Management 


Frequency of Usage 


Leadership (MLE) 


Student Progress 


Training and Support 


Teacher Buy-in 


Below Expectation 
0-1.99 


Teachers disagree or 
strongly disagree that they 
use data to inform instruction. 


Teachers disagree or 
strongly disagree that they 
are able to manage IM site to 
meet their needs. 


Teachers on average never 
review overview or standards 
reports pages in an academic 
year. 


Teachers disagree or 
strongly disagree that 
administrator's encourage 
integration, provide 
supports, and work with staff 
frequently. 


Students attempted 0 lessons 
in an academic year. 


Teachers disagree or 
strongly disagree that the 
vendor and trained Lead Staff 
provide training and support. 


Teachers disagree or 
strongly disagree that 
Imagine Math is beneficial to 
them and their students. 
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Approaching Expectation 
2.00—2.99 


Teachers agree or neutral 
that they use data to inform 
instruction. 


Teachers agree or neutral 
that they are able to manage 
IM site to meet their needs. 


Teachers on average review 
overview or standards reports 
monthly in an academic year. 


Teachers agree or neutral 
that administrator's encourage 
integration, provide supports, 
and work with staff 
frequently. 


Students attempted 1 to 29 
lessons in an academic year. 


Teachers are neutral or 
agree that the vendor and 
trained Lead Staff provide 
training and support. 


Teachers agree or neutral 
that Imagine Math is 
beneficial to them and their 
students. 


Met Expectation 
3.00—4.00 


Teachers strongly agree that 
they use data to inform 
instruction. 


Teachers strongly agree that 
they are able to manage IM 
site to meet their needs.. 


Teachers on average review 
overview or standards reports 
weekly in an academic year. 


Teachers strongly agree that 
administrator's encourage 
integration, provide supports, 
, and work with staff 
frequently. 


Students attempted 30 or 
more lessons in an academic 
year (Think 30). 


Teachers strongly agree that 
the vendor and trained Lead 
Staff provide training and 
support. 


Teachers strongly agree that 
Imagine Math is beneficial to 
them and their students. 
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APPENDIX-—C 


Data Sources for Imagine Math Implementation Indicators 


Item Description 


Index Score 


Standards-Based Score 


Data Analysis 
Cronbach's alpha=.899 


Data Usage 


Cronbach's alpha=.830 


Teacher 
Survey 


Teacher 
Survey 


Please indicate how each of the following statements 
apply to you regarding the use of Imagine Math® 
Reports 

(Q17)I am able to identify and address the needs of 
low standards and struggling students. 

(Q18) Lapply intervention and extension lessons 
based on the data. 

(Q19) [help the students to create goal(s) based on 
the report data. 

(Q20) I adjust the groupings / settings, usage 
frequency, instructional pathway, or journaling. 


Please indicate to what extent you agree or disagree 
with the following statements regarding your ability 
to use Imagine Math® to do the following: 

(Q21) Retrieve individual or group reports. 

(Q22) Identify areas where I need to adjust my 
instruction based on the reports retrieved. 

(Q23) Retrieve reports that best meet my needs. 

(Q24) Manipulate the program to meet the math needs 
of individual learners. 


4-point scale Z 
score 


5-point scale Z 
score 


0=Do not review 
standards 

1=Monthly 

2=Once per week 
3=More than once per 
week 


0=Strongly Disagree 
1=Dis agree 
2=Neutral 

3=A gree 

4=Strongly Agree 


Frequency of Usage 


Leadership 
Cronbach's alpha=.707 


on 
co 
2 
I 
s 
a 
a 
= 
3 
an 
r= 
is} 
3S 
2 
Ss 
° 
Lond 
Oo 


Teacher 
Survey 


Teacher 
Survey 


IM Database 


IM Database 


Teacher 
Survey 


Teacher 
Survey 


Q6. How often in the 2018 —2019 academic year have 
you used Imagine Math to inform your instruction? 


Q9. How frequently do you review each data source? 
a) Overview Reports 

b) Standards Reports 

c) Weekly Progress Reports 

d) Benchmark Reports 

e) Snapshot Reports 


Number of logins 


Time on IM website 


Please indicate to what extent you agree or disagree 
with the following statements: 

(Q27) School administrators have supported the use of 
Imagine Math® in the classroom. 

(Q31) Ican determine how best to use Imagine Math® 
in my classroom. 

(Q33) My students have access to computers to use 
Imagine Math® in my school. 


(Q4) How frequently in the 2018 — 2019 academic year 
has leadership worked with you to discuss student 
data (PLC) from Imagine Math®? 
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6-point scale Z 
score 


4-point scale Z 
score 


Continuous 
variable -.77 to 3.49 


Z score 
Continuous 
variable -.76 to 3.53 
Z score 

5-point scale Z 
score 


5-point scale Z 
score 


O=Never 

1=As Needed 

2=3-4 times a month 
3=2 times a month 
4=3 to 4 times a week 
5=1 to 2 times a week 


O=Never 
1=Monthly 
2=Weekly 
3=Daily 


0=Strongly Disagree 
1=Disagree 
2=Neutral 

3=A gree 

4=Strongly Agree 


O=Never 

1=As Needed 
2=Weekly 
3=Every 6-weeks 
4=Monthly 
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APPENDIX-C, continued 


Source 


Total Lessons Attempted 
ie oe 


Please indicate to what extent you agree or disagree 
with the following statements relating to training 
and support for Imagine Math®. 

(Q25) Imagine Learning Customer Success Managers 
have provided ongoing training and support. 

(Q26) The designated Imagine Math® Lead at my 
school is helpful (teachers or administrator that are 
trained to support other teachers ). 

(Q28) The one-on-one training received from Imagine 
Math® staff helped me improve my math instruction to 
students. 

(Q29) The Imagine Math® Quick Guide is a helpful 
resource for setting—up and using the software. 


Please indicate to what extent you agree or disagree 


Indicator Item Description 


Student Usage / 
Progress 


—_ 
=) 
—) 
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=) 

DN 

= 
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I 

= 
xs 
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Cronbach's alpha=.858 


with following statements relating to the usefulness of 
Imagine Math®. 

(Q30) Imagine Math® helps me to identify different 
areas of professional development that I need. 

(Q32) Imagine Math® helps me to reflect on my own 
teaching practices. 

(Q34) My students have access to computers to use 
Imagine Math® in my school. 

(Q35) Students are engaged and interested in the 
Imagine Math® Program. 

(Q36) Imagine Math® is beneficial to students to 
increase their math performance. 


Teacher Buy-In 
Cronbach's alpha=.917 


Index Score 


Continuous 
variable -.61 to 


19.73 Z score 
Continuous 


variable -.56 to 


31.91 Z score 
Continuous 


variable -.53 to 
23.81 Z score 
5-point scale Z 
score 


5-point scale Z 
score 


Standards-Based Score 


0=Strongly Disagree 
1=Disagree 
2=Neutral 

3=A gree 

4=Strongly Agree 


0=Strongly Disagree 
1=Dis agree 
2=Neutral 

3=A gree 

4=Strongly Agree 


Outcome 


Chancery Ad 


oe Performance on 3—8 STAAR or Algebra I EOC 


Note: Cronbach's Alpha: >.9 (Excellent), >.8 (Good), .7 (Acceptable), .6 (Questionable), > .5 (Poor), and < .5(Unacceptable) 
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APPENDIX-D 


Table D1. Hierarchal Regression Models Predicting the Effects of Imagine Math® on HISD Middle School 


Students’ Academic Achievement 


Model 1 Model 2 
B SE B t B SE B t 
(Constant) 486.79 43.44 11.21 502.21 44.95 11.17 
Scale Score STAAR 2017-2018 0.74 0.02 0.71% ** 31.20 0.73 0.03 0.70*** 29.68 
Gender 7.33 5.27 0.03 1.39 8.61 5.22 0.03 1.65 
[Male] 
Economically Disadvantaged 1.77 7.85 0.00 0.23 -0.80 7.93 0.00 -0.10 
At-Risk -22.91 6.55 -0.08*** — -3.50 -25.28 6.52 -0.08*** — -3.88 
GT 39.24 6.91 O.12*** = 5.68 38.54 6.85 O.11*** 5.63 
[0 is reference group] 
Home Language 
English -21.88 11.85 -0.07 -1.85 -21.18 11.73 -0.07 -1.80 
Spanish -24.28 13.68 -0.08 -1.78 -20.37 13.59 -0.07 -1.50 
[Other] 
Ethnicity 
Black -25.67 10.53 -0.07** -2.44 -30.62 10.58 -0.84* -2.89 
Hispanic -10.58 10.83 -0.04 -0.98 -17.78 10.80 -0.06 -1.65 
[White] 
Lessons Attempted (>30) 29.82 12.32 0.07** 2.42 
Lessons Attempted (<30) 10.30 9.67 0.03 1.07 
[Lessons Attempted =0] 
Points Earned 25.40 13.50 0.04 1.88 
[PointsEarned =0] 
Data Management 3.60 9.89 0.01 0.36 
Data Analysis -20.42 7.00 -0.06* -2.92 
Frequency of Use -26.16 6.46 -0.078** — -4.05 
[0 is reference group] 
R’ 0.711 0.718 
F 275.64 5.24 
*p<.05 **p<.0l ***p<.000 


DV 2019 STAAR 3-8 Mathematics. Students who took Algebra I in middle school not included. 
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Table D2. Hierarchal Regression Models Predicting the Effects of Imagine Math® on HISD High School 
Students’ Academic Achievement 


Model 1 Model 2 
B SE p t B SE p t 
(Constant) 391.32 222.15 1.76 361.76 239.17 1.51 
Scale Score STAAR 2017-2018 2.31 0.12 Pe as 19.05 2.21 0.12 0.51*** = 18.75 
Gender 139.96 26.00 14*** 5.38 110.92 25.34 0.10*** 4.38 
[Male] 
Economically Disadvantaged -41.82 41.35 -.03 -1.01 -86.86 40.48 -0.06 -2.15 
At-Risk -130.67 35.39 -1.07***  -3.69 -110.97 34.14 -0.91** = -4.17 
GT 140.12 55.18 O7** 2.54 158.61 53.47 0.08*** 2.46 
[0 is reference group] 
Home Language 
English -221.47 72.16 -.22*** — -3.07 -153.92 70.54 -0.15** — -2.18 
Spanish -182.50 79.04 -.18* -2.31 -126.21 77.11 -0.12* -1.64 
[Other] 
Ethnicity 
Black 19.57 67.13 -.02 -0.30 -55.76 65.42 -0.05 -0.85 
Hispanic -71.01 69.77 -.07 -1.02 -135.87 68.53 -0.13 -1.01 
[White] 
Lessons Attempted (>30) 270.93 51.11 0.19*** 5.30 
Lessons Attempted (<30) 71.40 39.45 0.06 1.81 
[Lessons Attempted =O] 
Points Earned 151.08 119.55 0.03 1.26 
[Points Earned =0] 
Data Management 133.33, 31.05 0.12*** — 4.30 
Data Analysis -91.16 30.79 -0.08* -2.96 
Frequency of Use -49.87 32.22 -0.05 -1.55 
[0 is reference group] 
R? 0.415 0.456 
F 73.97 14.26 


*D<.05 **p<.0l ***p<.000 
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